feat: index each language separately, and assert it stays that way - #447
Merged
Conversation
lens0021
added a commit
that referenced
this pull request
Aug 17, 2026
Split out of #447, which verified it. This is the half that can land now; the nightly pin and the smoke assertion that goes with it stay there, since they need each other and both wait on a formal SifterSearch release. ## The bug Pagefind names each language's index in one JSON map in `pagefind-entry.json` and writes that map in whatever order it iterated, which is not the same order twice. Two bakes of one source then produce two spellings of the same fact — identical index hashes and page counts, different language order: ``` < {"languages":{"ko":{"hash":"ko_72ba3bbda7",…},"en":{"hash":"en_e66da688eb",…},"km":{…}},…} > {"languages":{"en":{"hash":"en_e66da688eb",…},"ko":{"hash":"ko_72ba3bbda7",…},"km":{…}},…} ``` **This site does not show it yet.** Every page is indexed as the wiki's content language today, so there is one language and no order to get wrong. It appears the moment translations are indexed in their own, which is what chaotic-ground/SifterSearch#69 does — a bake against that build holds three (en 39, ko 19, km 1) and #411's two-bake check fails on this one file, in the root bundle and in both skin copies of it. ## Why it lands before the pin moves So the bump that brings the language fix in just passes. Otherwise that PR fails the reproducibility check for a reason that has nothing to do with what it changed, and whoever opens it — updatecli, most likely — gets to work out why. ## Where the fix belongs Reproducibility is this build's promise (#411), not the indexer's, so the build keeps it. That is the same work `StripBuildStamps` already does for the per-request ids MediaWiki leaves in a page: something upstream writes a value that varies per build, and the build settles it. Key order carries no meaning in JSON and the client looks a language up by name, so sorting the map changes nothing any reader of the bundle can tell. `Build::stabilizeSearchIndex()` runs in the orchestrator, after the index job `runJobs()` holds back to the end and before the skin passes, so the one bundle they all copy is already settled — nothing to do per pass. ## What the re-encode is careful about It does not try to reproduce Pagefind's bytes, only the same bytes every time, which is all the promise asks — so a change in Pagefind's own formatting cannot quietly break this. The flags keep it close anyway, so a diff against an older bake stays readable; escaping the `include_characters` list to `‿` would make the file differ for no reason, which is the very thing being fixed. An entry file that cannot be parsed, or that holds no language map, is returned as `null` and left exactly as Pagefind wrote it. A bundle this does not understand is not one to rewrite. ## Tests `Search::stableIndexEntry()` is pure, so `SearchTest` covers it directly. The fixtures are the real two files, off one runner, from one source — and the test asserts they differ before asserting they normalise to the same thing, so it cannot pass vacuously. `typos` reads the `ba` in a Pagefind hash (`ko_72ba3bbda7`) as a misspelling of `by`, so `.typos.toml` learns their shape, beside the entry already there for Translate's own hex stamps. ## Verification Proven in CI on #447, which pins the nightly carrying SifterSearch#69: the bake produced `en`, `ko` and `km` indexes, and **the two bakes came out identical** — the step that failed before this fix. On this branch, with the pin still at `v0.7.1`, the change is a no-op in effect: one language, nothing to reorder. --- _Generated by [Claude Code](https://claude.ai/code/session_01NiyBN88bnGhcXVUXkpD7Mc)_ Co-authored-by: Claude <noreply@anthropic.com>
Picks up SifterSearch v0.7.2, whose fix indexes each page in the language it is written in rather than the wiki's content language (chaotic-ground/SifterSearch#69). Pagefind builds one index per language it finds and its client selects by the language of the page the reader is on, so until now this site shipped one English index holding all 59 pages: Korean and Khmer were tokenised and stemmed by English rules, and a reader searching from a translated page was answered out of the English index because there was no other to fall back to. A bake now produces three -- en 39, ko 19, km 1. The smoke test says so out loud and fails if English and Korean are not indexed apart. It is the assertion this site was missing: nothing here would have caught a search answering a Korean reader out of an English index, because the search still worked. The existing Pagefind checks assert a bundle exists per skin copy (#399); none of them looked at what was in it. Khmer shows in the printed counts without being required, one page being a thin basis to gate on. Reproducibility is already handled: three languages made Pagefind's language map come out in a different order each bake, which #451 settled ahead of this so the bump would not fail the two-bake check for a reason unrelated to it. Verified before this pin moved, against the nightly carrying the same fix. Closes #400. --- _Generated by [Claude Code](https://claude.ai/code/session_01NiyBN88bnGhcXVUXkpD7Mc)_ Co-authored-by: Claude <noreply@anthropic.com>
lens0021
force-pushed
the
claude/pr-review-merge-knsiy3
branch
from
August 17, 2026 11:20
2b79c18 to
5c08445
Compare
lens0021
marked this pull request as ready for review
August 17, 2026 11:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps SifterSearch to v0.7.2 and adds the assertion that proves what it fixes. Rebased onto the current
main; the reproducibility half that was on this branch went in separately as #451.Closes #400 — but only its language half. The other half is filed as #454; see the end of this description.
What changes for a reader
SifterSearch indexed every page as the wiki's content language, whatever it was written in (chaotic-ground/SifterSearch#69). Pagefind builds one index per language it finds and its client picks the index by the language of the page the reader is on, so this site shipped one English index holding every page:
Two consequences. Korean and Khmer were tokenised and stemmed by English rules, so relevance for a translated page was whatever English stemming happened to give it. And a reader searching from a translated page was answered out of the English index — the client looked for
ko, found none, and fell back to the only one there was.A bake now produces three indexes: en 39, ko 19, km 1.
Searching from a page now answers in that page's language. Content not translated into the reader's language is no longer surfaced to them, which is the intended trade and the reason per-language indexes exist.
The assertion
smokeprints what the bundle holds and fails if English and Korean are not indexed apart:This is the check the site was missing. Nothing here would have caught a search answering a Korean reader out of an English index, because the search still worked — Pagefind falls back to whatever index exists. The existing Pagefind checks assert a bundle exists per skin copy (#399); none of them looked at what was inside it.
Khmer appears in the printed counts without being required to pass; one page is a thin basis to gate on.
Reproducibility, already handled
Three languages made Pagefind's language map come out in a different order each bake, breaking #411's byte-identical check on
pagefind-entry.jsonalone. #451 settled that ahead of this bump, so this PR does not fail the two-bake check for a reason unrelated to what it changes.How this was verified
The whole chain was proven against
nightly-2026-08-17before the pin moved to the release:<html lang>BuildIndexJobemits — 1 index when all pages claimen, 2 when they tell the truthgetPageLanguage()returns the translation's language for Translate subpagesen/ko/kmWhat #400 asked for that this does not do
#400 raised two faults. This closes the first and leaves the second, now tracked as #454: a translatable page reaches the export twice in the same language — as the source page and as the
/entranslation Translate materialises — so the English index holds 39 entries for 20 pages, and an English reader is offered each one twice.No language split can separate those two; they are the same language by construction. It needs a decision about what to index, and an extension point in SifterSearch to express it, since
SifterSearchNamespacescannot tell a source page from its translation.